On a Quantitative Version of the Oppenheim Conjecture

The Oppenheim conjecture, proved by Margulis in 1986, states that the set of values at integral points of an indefinite quadratic form in three or more variables is dense, provided the form is not proportional to a rational form. In this paper we study the distribution of values of such a form. We show that if the signature of the form is not (2, 1) or (2, 2), then the values are uniformly distributed on the real line, provided the form is not proportional to a rational form. In the cases where the signature is (2, 1) or (2, 2) we show that no such universal formula exists, and give asymptotic upper bounds which are in general best possible. Let Q be an indefinite nondegenerate quadratic form in n variables. Let LQ = Q(Z) denote the set of values of Q at integral points. The Oppenheim conjecture, proved by Margulis (cf. [Mar]) states that if n ≥ 3, and Q is not proportional to a form with rational coefficients, then LQ is dense. The Oppenheim conjecture enjoyed attention and many studies since it was conjectured in 1929 mostly using analytic number theory methods. In this paper we study some finer questions related to the distribution the values of Q at integral points. 1. Let ν be a continuous positive function on the sphere {v ∈ R | ‖v‖ = 1}, and let Ω = {v ∈ R | ‖v‖ < ν(v/‖v‖)}. We denote by TΩ the dilate of Ω by T . Define the following set: V Q (a,b)(R) = {x ∈ R n | a < Q(x) < b}. We shall use V(a,b) = V Q (a,b) when there is no confusion about the form Q. Also let V(a,b)(Z) = V Q (a,b)(Z) = {x ∈ Z | a < Q(x) < b}. The set TΩ ∩ Z consists of O(T) points, Q(TΩ ∩ Z) is contained in an interval of the form [−μT , μT ], where μ > 0 is a constant depending on Q and Ω. Thus one might expect that for any interval [a, b], as T →∞, |V(a,b)(Z) ∩ TΩ| ∼ cQ,Ω(b− a)Tn−2, (1.1) Received by the editors December 6, 1995. 1991 Mathematics Subject Classification. Primary 11J25, 22E40. Research of the first author partially supported by an NSF postdoctoral fellowship and by BSF grant 94-00060/1. Research of the second author partially supported by NSF grants DMS-9204270 and DMS9424613. Research of the third author partially supported by the Israel Science foundation and by BSF grant 94-00060/1. 1The full version is available electronically at http://www.math.uchicago.edu/~eskin c ©1995 American Mathematical Society